The present invention relates generally to processing of digital images, to erase a feature from a digital image.
A common task in the manipulation of digital images is the removal of one or more objects from a scene. Simple examples of this include: the removal of one or more background objects, leaving just a foreground object(s); the removal of a foreground object from a background scene; and, for composited images produced from multiple layers of image data, the removal of an object belonging to an original layer from the overall composite. The removal process is typically a difficult task for several reasons. A pixel at an edge of an object may have contributions from both the foreground and the background. The color of the pixel is consequently a blend of the two regions, making the removal of only one of the regions difficult. Even for an object that does not include blend data at the object edge (e.g., hard edges), the object border often contains detail that requires tedious effort to define manually, making the removal of the object difficult. Objects that include hair or fur present a combination of these problems. These types of objects have complex shapes and typically include regions with thin fibers that result in color blending.
In general, the problem does not have a simple unambiguous solution. The movie industry has adopted a solution that includes simplifying the scene, filming objects or people against a simple background (e.g., a blue screen) having a uniform color. However, even with simplified scenes, the removal of the background from complex objects remains a difficult process.
Techniques have been developed to produce approximate solutions. A software mask can be applied to an object to clip the object from the background. However, a haloing effect at the edge of the object will often result. The haloing effect can be minimized by manually tracing an object outline and closely cropping the mask to the outline, but the solution is tedious for complex objects, such as objects with hair. Alternatively, the mask can be sized to clip a portion of the foreground image away to remove pixels that include blend data. Neither of these solutions is satisfactory.
Another solution is to erase the undesirable portions of the image. There are various techniques for processing a raster image to erase components of the image. A common technique is to use a brush that is sized as one or more pixels in one or more dimensions. By positioning the brush over the image, the pixels that are covered by the brush are erased by one of two conventional erasing techniques. In a first technique, the erased colors are replaced with a background color. In a second technique, colors are effectively erased by driving their alpha values toward zero (where a full erasure occurs if the alpha channel value is set to zero).
Both of these techniques are effective for removing an undesirable portion of an image. However, the techniques also require tedious manual manipulations when operating on complex objects. In addition, conventional erasure techniques will not produce the correct original colors for pixels at object edges that include blend data. In the first technique, pixels that are part of an object edge and include blend data can either be erased completely (substituting a background color) or left un-erased. In either case, the results are similar to the haloing or close cropping problems described above. In the second technique, the color for blend pixels is unaffected, again resulting in undesirable results.
Transparency is a visual blending effect obtained when a background object (image, text, lines or filled regions) is partially obscured by a foreground object that is drawn over the background object. Numerous other blending effects may be used to integrate foreground and background graphics data. Examples of blending effects include a drop shadow effect, a screen effect, darker and lighter effects and overprinting effects.
A common task in the manipulation of digital images is the removal of a background object from a transparent foreground object. The transparent nature of the foreground object results in a combination of the foreground object blending with one or more background objects in the scene. In this example, the entire foreground object is a blend. It is difficult to erase one without erasing parts of the other. Typically, a conventional erasure algorithm is provided for erasing the background feature where alpha values that reflect the portion of the foreground element are extracted. However, even if the alpha values that reflect the amount of the foreground element that was present in the initial image are extracted, the color values remain a blend of the foreground and background features.
In one aspect, the invention provides a method for erasing a feature from a digital image and includes identifying a plurality of tip regions on the image each including multiple pixels of the image and each covering a portion of the feature and sequentially processing the tip regions to erase the feature. The sequential processing includes determining an erasure color, defining a color match function for the tip region, applying the color match function to each pixel in the tip region including determining a color match value for each pixel and completely erasing ones of the pixels in the tip region that have a color match value greater than a threshold value. For a pixel having a color match value less than the threshold, an erasure function is applied to the target pixel. A contribution of the erasure color is removed from a color associated with the target pixel where the effect of each action of processing a tip region is recorded in the digital image before any subsequent action of processing a tip region is performed.
Aspects of the invention include numerous features. The method can include applying a stroke using a brush to the digital image to designate a portion of the digital image for processing. The step of determining an erasure color can include sampling a color at a center region of a current brush tip of the brush stroke. The step of determining an erasure color can include sampling a color at a beginning of a brush stroke. The step of determining a background color can designate a fixed color for the erasure color.
The method can include adjusting a shape of the brush to size a tip region such that the stroke includes a portion of a boundary defining the intersection of the foreground and the background images. The shape of the brush can be a circle and the step of adjusting the shape for the brush can include varying a radius of the circle.
The step of erasing can include changing alpha values associated with a given pixel. The step of determining a color match value can include determining a difference between the erasure color and the color of the target pixel and applying a function to the difference. The function can be a tent function.
The method can include pinning the color match value to reflect colors available in a color space for the digital image. The threshold can be adjustable at each tip application. The color match value can be adjusted by a density value for the brush. The color match value can be reduced by a scaling factor proportional to a distance of the given target pixel from a center of a tip region.
The color match value at a target pixel can be driven to a minimum value if a target pixel between the current target pixel and a center of a tip region has color match value that is the minimum value. The color match value at a target pixel can be driven to a minimum value if all paths between a center of a tip application region and the target pixel include a pixel with a color match value that is the minimum value.
The image includes data in a plurality of color planes and a color match value is determined for each color plane. A minimum color match value for all the color planes can be used to determine whether or not a target pixel is completely or partially erased.
The erasure color and the color of a target pixel can include alpha channel values. The method can include determining a color match value from the alpha channel values.
The color values of the erasure color and the target pixels can be in a first color space. The method can include changing color values for the erasure color and the color of the target pixel to a second different color space prior to determining a color match value. The method can include changing color values for the erasure color and the color of the target pixel to a second different color space prior to removing a contribution of a background color.
The method can include changing color values for the erasure color and the color of the target pixel to a second different color space prior to determining a color match value and where the step of removing a contribution of a background color includes converting the decontaminated color values back to the first color space. The erasure color can be a background color.
In another aspect the invention provides a method for erasing a feature from a digital image and includes identifying a plurality of tip regions on the digital image each covering a portion of the feature, defining an erasure color, the erasure color being variable and potentially changing with each tip region and processing the tip regions to erase the feature. The step of processing the tip regions includes determining an erasure color for the tip region, applying an erasure function to each pixel in the region to remove the feature from the pixel and decontaminating each pixel in the region including removing a contribution of the erasure color from the pixel color if the pixel color is distinct from the erasure color.
In another aspect, the invention provides a method for erasing a feature from a digital image and includes identifying a plurality of tip regions on the digital image each covering a portion of the feature, defining a color match function to be applied to pixels in a tip region, the color match function being variable and potentially changing with each tip region and processing the tip regions to erase the feature. The processing includes determining an erasure color for the tip region, applying an erasure function to each pixel in the region to remove the feature from the pixel and decontaminating each pixel in the region including applying the color match function for the tip region to the pixel to remove a contribution of the erasure color from the pixel color if the pixel color is distinct from the erasure color.
In another aspect, the invention provides a method for erasing a feature from a digital image and includes identifying a plurality of tip regions on the digital image each covering a portion of the feature to be erased and processing the tip regions sequentially to erase the feature. The processing includes applying an erasure function to each pixel in the region to remove the feature from a target pixel where the action of applying the erasure function is given effect in the image before subsequent tip regions are processed and decontaminating each pixel in the region including removing a contribution of an erasure color from the pixel color if the pixel color is distinct from the erasure color.
Aspects of the invention can include numerous features. The step of applying the erasure function can include applying the erasure function at 100 percent to all pixels in the region having a color approximately equal to the erasure color, determining a color difference for all remaining pixels in the region and applying the erasure function at zero percent to all pixels having a color difference above a threshold value.
In another aspect, the invention provides a user interface method for processing a digital image and includes providing an erasure brush, applying the erasure brush in a stroke along a user-defined path covering a portion of a feature to be erased, selecting an erasure color at each tip application of the erasure brush as the erasure brush moves along the path and, at each tip application, invoking an erasing engine. The erasing engine applies a color match function to pixels lying within a tip region defined by the tip application, completely erases ones of the pixels in the tip region having a color that is approximately equivalent to the erasure color for the tip application, partially erases remaining ones of the pixels in the tip region having a color distinct from the erasure color, and decontaminates the color of partially erased pixels including removing a contribution of the erasure color from a color associated with the partially erased pixel.
Aspects of the invention include numerous features. The erasure brush can include a user adjustable geometry for defining a number of pixels enclosed in a tip region. The method can include configuring the erasure brush in response to a user input to define the geometry of the tip region processed at each tip application.
In another aspect, the invention provides a user interface for processing a digital image. The digital image includes a plurality of features represented by raster data. The user interface is manipulatable by a user to cause the removal of a feature from the digital image. The user interface includes a user adjustable brush defining a tip region for use in identifying a region of the digital image to process. The brush has a user specified shape. A means for adjusting the shape of the brush in accordance with user input is included. Additionally, means for moving the brush along a user defined path, means for selecting an erasure color at each tip application of the brush as the brush moves along the path and means for invoking an erasing engine at each tip application are included. The erasing engine applies a color match function to pixels lying within a tip region defined by a tip application, completely erases ones of the pixels in the tip region having a color that is approximately equivalent to the erasure color for a given tip application, partially erases remaining ones of the pixels having a color distinct from the erasure color, and decontaminates the color of a given pixel including removing a contribution of the erasure color from a color associated with a partially erased pixel.
Advantageous implementations of the present invention can include one or more of the following advantages. Background objects can be extracted from complex objects in complex scenes. A user interface is provided that enables quick and simple selection of an object to be erased. The user has the flexibility to highlight only those portions of the object that need to be erased. The user also has the flexibility to modify the brush geometry to capture the portion of the digital image that needs to be erased. Background objects of digital images can be extracted with high accuracy. The background surrounding a complex foreground object can easily be erased while avoiding halos of the background color in the edge pixels.
These and other advantages of the present invention will become apparent from the following description and from the claims.